Recent experimental results have pushed the limits of magnetization dynamics to pico- and femtosecond timescales. This ultra-fast spin dynamics occurs in extreme conditions of strong and rapidly varying fields and high temperatures. This situation requires new description of magnetization dynamics, even on a phenomenological level of the atomistic Landau-Lifshitz-Gilbert equation, taking into account that the correlation time for electron system could be of the order of the inverse characteristic spin frequency. For this case we introduce the thermodynamically correct phenomenological approach for spin dynamics based on the Landau-Lifshitz-Miyasaki-Seki equation. The influence of the noise correlation time on longitudinal and transverse magnetization relaxation is investigated. We also demonstrate the effect of the noise correlation time on demagnetisation rate of different materials during laser-induced dynamics.